Papers by Ayush K Chakravarthy
Systematicity Emerges in Transformers when Abstract Grammatical Roles Guide Attention (2022.naacl-srw)
Copied to clipboard
| Challenge: | Existing systems that use transformers lack systematicity, but they are inferior to human learners in sample efficiency and difficult generalization problems. |
| Approach: | They propose to modify a transformer so that it controls attention distributions and fills in the gaps. |
| Outcome: | The proposed model shows that the performance of natural language processing systems is improved when abstract role labels are assigned to the input stream and provided to the role stream. |